Click any tag below to further narrow down your results
Links
The first part of a series where the author writes data governance rules as code, builds a platform to test them, and has CI block any change that breaks a policy. It walks through how the data mesh principles map onto folders and checks.
- Governance rules live as files, and CI runs them on every pull request: a partner file that doesn't match its contract never reaches the warehouse, and a change that breaks another team's report fails and names that team.
- The project is organized around Zhamak Dehghani's four data mesh principles, mapped to folders (domains/, data_products/, platform/, governance/), with five domains and exactly one owner per dataset.
- The author favors end-to-end "proofs" over unit tests, since a proof checks that a policy still holds when all the pieces run together.
Data mesh hype cooled, but the underlying need for findable, trusted, owned data hasn't gone away. The author argues data products are shifting from a full enterprise restructuring into a practical discipline built into existing platforms, catalogs, and governance work.
- Data mesh asks for a big operating model change, and adoption stalled: a 2024 BARC study of 197 professionals found 20% had it underway and 24% planned it, down from 35% planning it in 2023.
- A data product is defined by a known consumer, a stated purpose, an accountable owner, and measurable use, not by the asset format itself.
- KPMG's 2025 survey of 250 executives found 92% saw data products as critical, yet only 35% reported extensive value, a gap between intent and execution.
- AI agents raise the stakes: without clear ownership, meaning, quality, and permitted use, they make weak decisions faster, so governed, portable product definitions matter more.
Grab built an automated system called Data Production Issues (DPIs) that detects when certified data products fail their contracts, diagnoses root causes without manual intervention, and fixes common failures automatically. The system keeps data quality issues actionable by filtering noise, assigning problems to the right owner, and closing issues only when the underlying data actually recovers.
- DPIs use a standardized Data Health API with four error categories (upstream, platform, job, data) to route failures to the correct team without requiring platform-specific knowledge, enabling consistent incident management across Grab's heterogeneous data systems.
- The triage phase deduplicates alerts and groups related failures by root cause, preventing responders from drowning in repetitive notifications while ensuring only real contract breaches become operational issues.
- Hugo, Grab's ingestion platform, consolidates signals from multiple sources into a three-layer diagnosis workflow that automatically identifies root causes and recommends fixes, eliminating manual log inspection and shortening remediation time.